Several technologies have recently emerged that allow users to experience a blend of reality and virtual worlds. For example, head-mounted display (HMD) devices may include high definition cameras, depth cameras, range finders, gyroscopes, accelerometers, and other technologies that allow the head-mounted HMD device to map the real world and to display on an at least partially see through display virtual objects head-mounted that may be viewed in conjunction with real objects that are visible through the display. This blended view of real-world objects and virtual objects is referred to as augmented reality.
One challenge associated with augmented reality systems is how to afford the user an ability to virtually modify the real environment through the creation and display of holographic content, in an intuitive manner In one particular example, the user may desire to modify the color of a real object. One prior approach involves obtaining geometric data defining the three dimensional shape of the real object, and then using that geometric data to generate a hologram to be displayed in a shape and location indicated by the geometric data, so as to appear to the user to color the real object. However, in many circumstances such three dimensional geometric data is not available, is inaccurate, and or is time consuming to process.